Skip to content

fix(tbench2_env): a missing reward.txt is a scoring error, not reward 0.0 - #1025

Merged
nblintao merged 2 commits into
huggingface:mainfrom
nblintao:tb2-no-verdict-on-missing-reward
Aug 29, 2026
Merged

fix(tbench2_env): a missing reward.txt is a scoring error, not reward 0.0#1025
nblintao merged 2 commits into
huggingface:mainfrom
nblintao:tb2-no-verdict-on-missing-reward

Conversation

@nblintao

@nblintao nblintao commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #1012, closing the last gap where an infrastructure failure could masquerade as a task failure (spotted by @Shi-Dong reviewing the downstream consumer).

What

When the canonical harness runs but no verdict is ever writtentests/test.sh crashed or was killed before its verifier wrote /logs/verifier/reward.txt (e.g. the uvx toolchain download failed, or the in-shell timeout fired) — evaluate previously returned reward 0.0 with the normal canonical-harness info, byte-for-byte indistinguishable from tests genuinely failing.

Now _parse_canonical_reward returns None for an absent/empty/non-numeric verdict and both callers (local + docker mode, via a shared _require_canonical_verdict) raise: step() reports it through its existing error path — observation.error set, reward=None, episode keeps its session — with the test.sh log tail in the message for diagnosis. Verify-window cleanup still runs (covered by tests in both modes).

Why it matters

RL consumers must be able to drop a no-verdict episode: ingesting it as 0.0 injects a false negative into training that no client-side check can catch (the reply looked completely valid). A genuine test failure still scores 0.0 — test.sh's verifier writes reward.txt = 0 in that case, so the only replies that now error are ones where the verifier never ran to completion.

Behavior change

Only the no-verdict edge case: evaluate replies that used to be a silent 0.0 are now an explicit scoring error. Genuine passes/failures and the pytest-fallback path (which scores by exit code and always has one) are unchanged.

Validation

pytest tests/envs/test_tbench2_env.py → 26 passed: two new tests (local + docker) assert an empty reward marker raises RuntimeError("produced no verdict") and that staged verifier assets are still wiped on the error path.

🤖 Generated with Claude Code


Note

Medium Risk
Changes only the no-verdict edge case for canonical harness scoring, but it alters evaluate observability for RL consumers and could surface previously silent infrastructure failures as errors.

Overview
When the canonical tests/test.sh harness runs but never writes a verdict (reward.txt missing, empty marker, or non-numeric value), scoring no longer returns 0.0 as if tests failed.

_parse_canonical_reward now returns None in those cases instead of coercing to 0.0. _require_canonical_verdict raises RuntimeError with a log tail so local and Docker canonical evaluate paths surface a scoring failure through step() (observation.error, reward=None) rather than a false negative for RL pipelines.

Genuine pass/fail (numeric reward.txt) and the pytest fallback path are unchanged. New unit tests cover the empty-marker case in both local and Docker modes and assert verifier staging still cleans up on error.

Reviewed by Cursor Bugbot for commit 84bcc90. Bugbot is set up for automated code reviews on this repo. Configure here.

… 0.0

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bot-ci-comment

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant